Visualising relevance criteria in exploratory search

In this article we present a set of approaches to aid the analysis of data gathered during experimentation with exploratory search systems and users acts of judging the relevance of the information retrieved by the system. We present three approaches: relevance criteria profile, relevance judgement complexity, and session visualisation. Relevance criteria profiles capture the prominence of each criterion usage with respect to the search sessions of individuals or selected user groups (e.g. groups defined by the users affiliations and/or level of research experience). We also examine relevance judgement complexity, that is, the number of criteria involved in a single judgment process. This allows the researcher to analyse the variations of complexities within and across search sessions. Finally, we propose a session visualisation technique that brings these results together and potentially allows the researcher to quickly detect emerging patterns with respect to interactions, relevance criteria usage, and complexity. The use of these tools is illustrated by analysing and discussing results from a user study that was conducted at the Robert Gordon University in 2008. We conclude by highlighting how the results of this study might be used Supported by the Engineering and Pysical Sciences Research Council as part of the project ”Automatic Adaptation of Knowledge Structures for Assisted Information Seeking (AutoAdapt)” [EP/F035705/1] U. Cerviño Beresi, Y. Kim, and D. Song The Robert Gordon University, School of Computing E-mail: [email protected], [email protected], [email protected] I. Ruthven The Strathclyde University, Department of Computer and Information Sciences E-mail: [email protected] to support the improvement of end-user services in digital libraries.